Converting Audio to Haptic Feedback in an Electronic Device

ABSTRACT

In general, in one aspect, a method performed by one or more processes executing on a computer systems includes receiving an audio signal comprising a range of audio frequencies including high frequencies and low frequencies, converting a first portion of the range of audio frequencies into haptic data, shifting a second portion of the range of audio frequencies to a different range of audio frequencies, and presenting at least one of the converted first portion and the shifted second portion to a human user. Other implementations of this aspect include corresponding systems, apparatus, and computer program products.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application Ser.No. 61/493,380, entitled “Audio Conversion To Vibration Patterns,” filedon Jun. 3, 2011, and U.S. application Ser. No. 13/231,381, entitled“Converting Audio to Haptic Feedback in an Electronic Device,” which arehereby incorporated by reference in their entirety.

TECHNICAL FIELD

This disclosure relates generally to shifting audio frequency ranges andconverting audio data into haptic data to convey information throughhaptic feedback to a user of a mobile device.

BACKGROUND

A person with a hearing impairment may be deaf to certain frequenciespotentially limiting the person's ability to fully realize informationcontained in audio data. For example, listening to music rich in highfrequencies may be less enjoyable to a person who is partially orcompletely deaf to some or all of those high frequencies.

Mobile devices typically include a mechanism for providing hapticfeedback. For example, a mobile phone may include a motor, e.g., apiezoelectric motor, for providing haptic feedback to a user of themobile device.

SUMMARY

This disclosure describes technology, which can be implemented as amethod, apparatus, and/or computer software embodied in acomputer-readable medium, to convert audio data to haptic data, forexample, for use in conveying audible information to a hearing-impaireduser of a mobile device through haptic feedback.

In general, in one aspect, a method performed by one or more processesexecuting on a computer systems includes receiving an audio signalcomprising a range of audio frequencies including high frequencies andlow frequencies, converting a first portion of the range of audiofrequencies into haptic data, shifting a second portion of the range ofaudio frequencies to a different range of audio frequencies, andpresenting at least one of the converted first portion and the shiftedsecond portion to a human user. Other implementations of this aspectinclude corresponding systems, apparatus, and computer program products.

This, and other aspects, can include one or more of the followingfeatures. The second portion of the range of audio frequencies may beshifted down to a lower range of audio frequencies. The performance ofthe converting and the shifting can overlap in time at least in part.The performance of the converting and the shifting can be orderindependent. The presenting may occur while one or more of thereceiving, converting and shifting are ongoing. Presenting the convertedfirst portion may comprise providing the human user with haptic feedbackvia a haptic mechanism associated with the electronic device and thehaptic feedback may comprise vibration, temperature variation, orelectric stimulus. Presenting the shifted second portion may compriseproviding the human user with sounds corresponding to the shifted secondportion via an audio output mechanism associated with the electronicdevice. The first portion and the second portion may be mutuallyexclusive. The first portion and the second portion may overlap at leastin part. Converting the first portion may comprise converting a subsetof the low frequencies to haptic data. Shifting the second portion maycomprise shifting a subset of the high frequencies to lower frequencies.The electronic device may comprise a mobile communications device havingan audio subsystem and a haptic subsystem. The one or both of theconverting and the shifting may be performed according to one or morehearing-related parameters associated with the human user where thehearing-related parameters associated with the human user may be definedby a hearing profile associated with the human user.

Potential advantages described in this disclosure may include improveddelivery of audible data to a hearing-impaired user of a mobile device.For example, a user of a mobile device may have a specific hearingimpairment that renders the user partially or completely deaf to acertain range of high frequencies. By shifting that particular range ofhigh frequencies into a frequency range audible to the hearing-impaireduser, and converting to a vibration pattern those frequencies in a lowerfrequency range that convey sound effect information, thehearing-impaired user may still be provided with the sensation ofenjoying the original information in the audio data.

Another potential advantage may include real-time audio frequencyshifting and conversion into haptic data. For example, a user of amobile device may be at a concert listening to music rich in highfrequencies. The user may desire to have the high frequencies shifted todifferent frequencies and/or converted into a vibration pattern toaugment the audible information. Such conversion can be done in areal-time manner at the mobile device and the information can beconveyed to the user through any suitable audio reproduction deviceand/or haptic mechanism.

Another potential advantage may include using a haptic mechanism of amobile device to alert a user of specific events. For example, a uniquevibration pattern may be assigned to any number of events such that theunique vibration pattern can alert the user of the specific event. Forexample, ambient noise can be detected by the mobile device, e.g., firealarms, cars, emergency vehicles, car horns, screams, dog barking,music, environmental noise, phone ringing, knock at the door, etc. Inresponse to detecting the ambient noise, a unique vibration pattern canbe actuated by a haptic mechanism within a mobile device to augment theauditory information with haptic feedback. In some implementations, adatabase of classified sounds may exist. Ambient audio data received bya mobile device may be compared with the database to determine aclassification of the ambient audio data and upon determining the typeof audio data the ambient audio data corresponds to, haptic feedback maybe provided to a user of the mobile device based on the classification.

Another potential advantage may include using haptic feedback inconnection with a musical instrument. For example, a hearing impaireduser may play a musical instrument and concurrently receive hapticfeedback from a mobile device that informs them user whether they areplaying the musical instrument in the correct tune or playing a songcorrectly.

Details of one or more implementations are set forth in the accompanyingdrawings and the description below. Other features, aspects, andpotential advantages will be apparent from the description and drawings,and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of exemplary architecture of a mobile device.

FIG. 2 is a block diagram of an exemplary network operating environmentfor mobile devices.

FIG. 3 is a block diagram of an exemplary mobile device configured toconvert audio data to vibration patterns.

FIG. 4 is a block diagram of an exemplary haptic subsystem.

FIG. 5 is a block diagram of playback of audio and haptic data.

FIG. 6 is a flowchart of an exemplary method for converting audio datato haptic data.

FIG. 7 is a flowchart of an exemplary method for converting audio datato haptic data.

FIG. 8 is a flowchart of an exemplary method for creating a hearingprofile.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

FIG. 1 is a block diagram of exemplary architecture 100 of a mobiledevice configured to perform haptic-based operations. A mobile devicecan include memory interface 102, one or more data processors, imageprocessors and/or processors 104, and peripherals interface 106. Memoryinterface 102, one or more processors 104 and/or peripherals interface106 can be separate components or can be integrated in one or moreintegrated circuits. Processors 104 can include one or more applicationprocessors (APs) and one or more baseband processors (BPs). Theapplication processors and baseband processors can be integrated in onesingle process chip. The various components in mobile device 100, forexample, can be coupled by one or more communication buses or signallines.

Sensors, devices, and subsystems can be coupled to peripherals interface106 to facilitate multiple functionalities. For example, motion sensor110, light sensor 112, and proximity sensor 114 can be coupled toperipherals interface 106 to facilitate orientation, lighting, andproximity functions of the mobile device. Motion sensor 110 can includeone or more accelerometers configured to determine change of speed anddirection of movement of the mobile device. Location processor 115(e.g., GPS receiver) can be connected to peripherals interface 106 toprovide geopositioning. Electronic magnetometer 116 (e.g., an integratedcircuit chip) can also be connected to peripherals interface 106 toprovide data that can be used to determine the direction of magneticNorth. Thus, electronic magnetometer 116 can be used as an electroniccompass. Gravimeter 117 can be coupled to peripherals interface 106 tofacilitate measurement of a local gravitational field of Earth.

Camera subsystem 120 and an optical sensor 122, e.g., a charged coupleddevice (CCD) or a complementary metal-oxide semiconductor (CMOS) opticalsensor, can be utilized to facilitate camera functions, such asrecording photographs and video clips.

Communication functions can be facilitated through one or more wirelesscommunication subsystems 124, which can include radio frequencyreceivers and transmitters and/or optical (e.g., infrared) receivers andtransmitters. The specific design and implementation of thecommunication subsystem 124 can depend on the communication network(s)over which a mobile device is intended to operate. For example, a mobiledevice can include communication subsystems 124 designed to operate overa CDMA system, a WiFi™ or WiMax™ network, and a Bluetooth™ network. Inparticular, the wireless communication subsystems 124 can includehosting protocols such that the mobile device can be configured as abase station for other wireless devices.

Audio subsystem 126 can be coupled to a speaker 128 and a microphone 130to facilitate voice-enabled functions, such as voice recognition, voicereplication, digital recording, and telephony functions. In someimplementations, audio subsystem 126 can be wirelessly coupled tospeaker 128. For example, audio subsystem 126 may be coupled to speaker128 using Bluetooth, WIFI, and the like. In some implementations,speaker 128 can be a hearing aid (e.g., a cochlea implant) wirelessly ordirectly coupled to audio subsystem 126. Microphone 130 may, forexample, be configured to detect ambient noise and other audiblefrequencies.

Haptic subsystem 180 and haptic mechanism 182, e.g., spinning motor,servo motor, piezoelectric motor, vibrator, etc., can be utilized tofacilitate haptic feedback, such as vibration, force, and/or motions. Inaddition, haptic mechanism 182 may be further capable of providing otherforms of haptic feedback. For example, haptic mechanism 182 may beconfigured to provide feedback in the form of variable temperatures(e.g., hot, warm, and cold) or electric stimulus.

I/O subsystem 140 can include touch screen controller 142 and/or otherinput controller(s) 144. Touch-screen controller 142 can be coupled to atouch screen 146 or pad. Touch screen 146 and touch screen controller142 can, for example, detect contact and movement or break thereof usingany of a plurality of touch sensitivity technologies, including but notlimited to capacitive, resistive, infrared, and surface acoustic wavetechnologies, as well as other proximity sensor arrays or other elementsfor determining one or more points of contact with touch screen 146.

Other input controller(s) 144 can be coupled to other input/controldevices 148, such as one or more buttons, rocker switches, thumb-wheel,infrared port, USB port, and/or a pointer device such as a stylus. Theone or more buttons (not shown) can include an up/down button for volumecontrol of speaker 128 and/or microphone 130.

In one implementation, a pressing of the button for a first duration maydisengage a lock of the touch screen 146; and a pressing of the buttonfor a second duration that is longer than the first duration may turnpower to mobile device 100 on or off. The user may be able to customizea functionality of one or more of the buttons. The touch screen 146 can,for example, also be used to implement virtual or soft buttons and/or akeyboard.

In some implementations, mobile device 100 can present recorded audioand/or video files, such as MP3, AAC, and MPEG files. In someimplementations, mobile device 100 can include the functionality of anMP3 player. Mobile device 100 may, therefore, include a pin connectorthat is compatible with the iPod. Other input/output and control devicescan also be used.

Memory interface 102 can be coupled to memory 150. Memory 150 caninclude high-speed random access memory and/or non-volatile memory, suchas one or more magnetic disk storage devices, one or more opticalstorage devices, and/or flash memory (e.g., NAND, NOR). Memory 150 canstore operating system 152, such as Darwin, RTXC, LINUX, UNIX, OS X,WINDOWS, or an embedded operating system such as VxWorks. Operatingsystem 152 may include instructions for handling basic system servicesand for performing hardware dependent tasks. In some implementations,operating system 152 can include a kernel (e.g., UNIX kernel).

Memory 150 may also store communication instructions 154 to facilitatecommunicating with one or more additional devices, one or more computersand/or one or more servers. Memory 150 may include graphical userinterface instructions 156 to facilitate graphic user interfaceprocessing; sensor processing instructions 158 to facilitatesensor-related processing and functions; phone instructions 160 tofacilitate phone-related processes and functions; electronic messaginginstructions 162 to facilitate electronic-messaging related processesand functions; web browsing instructions 164 to facilitate webbrowsing-related processes and functions; media processing instructions166 to facilitate media processing-related processes and functions;GPS/Navigation instructions 168 to facilitate GPS and navigation-relatedprocesses and instructions; camera instructions 170 to facilitatecamera-related processes and functions; magnetometer data 172 andcalibration instructions 174 to facilitate magnetometer calibration. Thememory 150 may also store other software instructions (not shown), suchas security instructions, web video instructions to facilitate webvideo-related processes and functions, and/or web shopping instructionsto facilitate web shopping-related processes and functions. In someimplementations, the media processing instructions 166 are divided intoaudio processing instructions and video processing instructions tofacilitate audio processing-related processes and functions and videoprocessing-related processes and functions, respectively. An activationrecord and International Mobile Equipment Identity (IMEI) or similarhardware identifier can also be stored in memory 150. Memory 150 caninclude haptic instructions 176. Haptic data 176 can be configured tocause the mobile device to perform haptic-based operations, for exampleproviding haptic feedback to a user of the mobile device as described inreference to FIGS. 2-8.

Each of the above identified instructions and applications cancorrespond to a set of instructions for performing one or more functionsdescribed above. These instructions need not be implemented as separatesoftware programs, procedures, or modules. Memory 150 can includeadditional instructions or fewer instructions. Furthermore, variousfunctions of the mobile device may be implemented in hardware and/or insoftware, including in one or more signal processing and/or applicationspecific integrated circuits.

FIG. 2 is a block diagram of exemplary network operating environment 200for the mobile devices configured to perform motion-based operations.Mobile devices 202 a and 202 b can, for example, communicate over one ormore wired and/or wireless networks 210 in data communication. Forexample, a wireless network 212, e.g., a cellular network, cancommunicate with a wide area network (WAN) 214, such as the Internet, byuse of a gateway 216. Likewise, an access device 218, such as an 802.11g wireless access device, can provide communication access to the widearea network 214.

In some implementations, both voice and data communications can beestablished over wireless network 212 and the access device 218. Forexample, mobile device 202 a can place and receive phone calls (e.g.,using voice over Internet Protocol (VoIP) protocols), send and receivee-mail messages (e.g., using Post Office Protocol 3 (POP3)), andretrieve electronic documents and/or streams, such as web pages,photographs, and videos, over wireless network 212, gateway 216, andwide area network 214 (e.g., using Transmission ControlProtocol/Internet Protocol (TCP/IP) or User Datagram Protocol (UDP)).Likewise, in some implementations, the mobile device 202 b can place andreceive phone calls, send and receive e-mail messages, and retrieveelectronic documents over the access device 218 and the wide areanetwork 214. In some implementations, mobile device 202 a or 202 b canbe physically connected to the access device 218 using one or morecables and the access device 218 can be a personal computer. In thisconfiguration, mobile device 202 a or 202 b can be referred to as a“tethered” device.

Mobile devices 202 a and 202 b can also establish communications byother means. For example, wireless mobile device 202 a can communicatewith other wireless devices, e.g., other mobile devices 202 a or 202 b,cell phones, etc., over the wireless network 212. Likewise, mobiledevices 202 a and 202 b can establish peer-to-peer communications 220,e.g., a personal area network, by use of one or more communicationsubsystems, such as the Bluetooth™ communication devices. Othercommunication protocols and topologies can also be implemented.

The mobile devices 202 a or 202 b can, for example, communicate with oneor more services 230, 240, and 250 over the one or more wired and/orwireless networks. For example, one or more hearing profile trainingservices 230 can be used to deliver one or more hearing profiles.Hearing profile delivery service 240 can provide one or more hearingprofiles to mobile devices 202 a and 202 b for converting audio data tohaptic data. Additionally, combined audio and haptic data deliveryservice 250 can provide one or more data files containing audio and/orhaptic data for playback at mobile devices 202 a and 202 b.

Mobile device 202 a or 202 b can also access other data and content overthe one or more wired and/or wireless networks. For example, contentpublishers, such as news sites, Really Simple Syndication (RSS) feeds,web sites, blogs, social networking sites, developer networks, etc., canbe accessed by mobile device 202 a or 202 b. Such access can be providedby invocation of a web browsing function or application (e.g., abrowser) in response to a user touching, for example, a Web object.

FIG. 3 is a block diagram of an exemplary mobile device 300 configuredto convert audio data to vibration patterns. Mobile device 300 caninclude a microphone 302, an audio data store 304, and input subsystem306. Microphone 302 can be configured to detect audible frequencies inreal-time, for example, ambient noise, music, talking, or any otheraudibly detectable frequencies. Audio data store 304 can include astorage device that stores one or more audio data files. Input subsystem306 can include line-in functionality to receive audio data from anotherdevice, for example another mobile device without the audio conversionto vibration functionality described in this disclosure.

Mobile device 300 can include audio subsystem 316 and haptic subsystem314. Audio subsystem 316 can be configured to receive and processesaudio data 308 received from any one of microphone 302, audio data store304, and input subsystem 306. Audio subsystem 310 can process audio data308 in accordance with a hearing profile 318 received from hearingprofile data store 312. For example, based on hearing profile 318, audiosubsystem 310 can shift a certain range of audible frequencies in audiodata 308 into a different range of audible frequencies. In someimplementations, audio profile 318 corresponds to the hearingcapabilities of a user of mobile device 300 and indicates that the useris deaf to a certain range of frequencies. For example, an elderly usermay have a specific hearing impairment that renders the elderly userpartially or completely deaf to a range of high frequencies. In suchcase, it may be difficult for the elderly user to enjoy audio data,e.g., music and other audible stimulations, containing high frequencieswithin that range. To accommodate such users with a hearing impairmentto a range of high frequencies, using corresponding hearing profile 318,audio subsystem 310 can shift a specific range of high frequencies inaudio data 308 into a lower range of frequencies that, according tohearing profile 318, the user can hear. For example, high frequencies inaudio data 308 that a user is unable to hear may be shifted into anaudible range of middle frequencies. The original middle frequencies canfurther be shifted into low frequencies, and the original lowfrequencies can be converted into a vibration pattern, discussed in moredetail below. Frequency shifting can be performed by any suitablemethod. For example, by a linear shift of frequencies, morphing thefrequencies, or applying common techniques involving Fourier Transformsto isolate a certain range of frequencies.

Haptic subsystem 314 can be configured to receive and process audio data308 received in real-time from microphone 302, from storage in audiodata store 304, or input from another source via input subsystem 306.Haptic subsystem 314 can process audio data 308 in accordance with ahearing profile 318 received from hearing profile data store 312. Forexample, based on audio profile 318, haptic subsystem 314 can convert arange of audible frequencies in audio data 308 into a vibration pattern324. Returning again to the hearing profile 318 of the elderly user witha specific hearing impairment that renders the elderly user partially orcompletely deaf to a range of high frequencies. The hearing profile 318may indicate that the elderly user, in addition to or in place offrequency shifting performed by audio subsystem 310, desires to have thespecific range of high frequencies converted into a vibration pattern324. Similarly, the hearing profile 318 may indicate that the elderlyuser desires a range of lower frequencies be converted into a vibrationpattern 324. In some implementations, a range of lower frequencies canbe converted into a vibration pattern directly in response to a user'shearing profile. Alternatively, a range of lower frequencies may beconverted into a vibration pattern as a result of down-shifting allaudio frequencies. That is, the high frequencies can be shifted tomiddle frequencies, the middle frequencies can be shifted to lowfrequencies, and the low frequencies can be converted into a vibrationpattern.

In other implementations, hearing profile 318 may correspond to thehearing preferences of a user of mobile device 300 and indicate that theuser prefers certain frequencies in audio data 308 to be converted fromaudible frequencies into a vibration pattern 324. For example,regardless of whether the user of mobile device 300 has a hearingimpairment, the user may desire to have a range of audible frequenciesin audio data 308 converted into a vibration pattern 324. For example,the user may be watching a motion picture and desire to receive hapticfeedback corresponding to sound effects in the motion picture. Althoughconverting audio data into vibration patterns may not produce anaccurate representation of the audio data, it may still provide a userwith the sensation of enjoying the audio data by conveying theinformation associated with the audio data to the user.

Audio subsystem 310 and haptic subsystem 314 may operate in conjunctionwith one another to convey information associated with audio data to auser of mobile device 300. For example, a user of mobile device 300 maybe watching a movie rich in high audible frequencies. In addition, theuser may possess a hearing impairment that renders the user partially orcompletely deaf to a range of higher frequencies corresponding totalking or dialog in the movie. Utilizing audio subsystem 308 and hapticsubsystem 314, although the user might have a hearing impairmentrelating to those higher frequencies, the user may still receive theinformation contained in those higher frequency ranges. For example,haptic subsystem 314 can convert the lower frequencies, e.g., soundeffects and explosions, into vibration patterns and audio subsystem 310can shift the higher frequencies that the user cannot hear into a lowerrange of frequencies which the user can hear.

In some implementations, sounds effects and spoken words are separatedout into separate tracks for a movie. In such a case, audio subsystem310 may be utilized to frequency shift all or a portion of the spokenwords track into a different frequency range. Similarly, hapticsubsystem 314 may be utilized to convert all or a portion of the soundeffects track to haptic data and a corresponding vibration pattern.

Mobile device 300 can include mixer 330 and combined audio and hapticdata store 344. Mixer 330 may be configured to receive audio data 322from audio subsystem 322 and haptic data 324 from haptic subsystem 314.Mixer 330 can combine audio data 322 and haptic data 324 into a file ofcombined audio and haptic data. Data combined by mixer 330 can be storedin combined audio and haptic data store 344 for later playback by amobile device.

Mobile device 300 may also include or be in communication with one ormore audio reproduction devices or haptic feedback devices, for examplespeaker(s) 340 and haptic mechanism(s) 342. Speaker(s) 340 may, forexample, be contained within mobile device 300. In some implementations,mobile device 300 may be in communication with headphones, copularimplants, external hearing aids, or a Bluetooth device comprisingspeaker(s) 340. Haptic mechanism(s) 342 may be any suitable device forproviding haptic feedback. For example, a spinning motor, servo motor,or piezoelectric motor, can be utilized to facilitate haptic feedback,such as vibration, force, and/or motions. In some implementations,haptic mechanism(s) 342 may be capable of providing haptic feedback inthe form of variable temperatures (e.g., hot, warm, cold) or electricstimulus.

FIG. 4 is a block diagram illustrating an exemplarily haptic subsystem400. Haptic subsystem 400 may be configured to receive both haptic data402 and audio data 404. In some implementations, haptic data 404 may bedata that has previously been processed from audio data into hapticdata. For example, haptic data 324 in FIG. 3. Audio data 404 may bereceived from microphone 302 in real-time, from storage in audio datastore 304, and/or input from another source via input subsystem 306.

Dynamic filtering subsystem 410 can perform operations on the hapticdata 402 and audio data 404 in accordance with a hearing profile 418received from hearing profile data store 412. For example, dynamicfiltering subsystem can perform a low pass filter on audio data 404 tofilter frequency ranges as specified in hearing profile 418. Likewise,dynamic filtering subsystem 410 can perform a band-pass or high-passfilter on audio data 404 to filter specific ranges of audio frequenciesas specified in hearing profile 418. In some implementations, dynamicfiltering subsystem 410 can perform a dynamic analysis of the audio data404 to determine which frequencies should be converted into haptic data.Any suitable technique for filtering audio data may be implemented bydynamic filtering subsystem 410.

Audio conversion to vibration pattern subsystem 430 may receive filteredaudio data from dynamic filtering subsystem 410 and convert a specificrange or ranges into a corresponding vibration pattern. For example, ifdynamic filtering subsystem 410 utilized a low-pass filter, audioconversion to vibration pattern subsystem may isolate intensitiescorresponding to the lower frequencies and create a suitable vibrationpattern to be stored as haptic data. In some implementations, thevibration pattern can be created by taking an average of a specificrange of frequencies, for example the range (20 hz-40 hz), to determinehow much intensity to be included in the vibration pattern. Similarly,dynamic filtering subsystem 410 can assign a weight to the most commonlow frequencies and base a vibration pattern on the occurrence of themost common low frequencies.

In a mobile device with multiple haptic mechanisms 444, 446, and 448,audio conversion to vibration pattern subsystem 430 can include a hapticmechanism selection subsystem 432 to determine at which haptic mechanisma vibration pattern should be actuated. Haptic mechanisms 444, 446, and448 may, for example, be any one of a spinning motor, servo motor,vibrator, piezoelectric device, or other suitable mechanical device forproviding haptic feedback. Each haptic mechanism 444, 446, and 448, maybe suitable for actuating haptic feedback corresponding to a certainaudio frequency range. For example, haptic mechanism 444 may be bestsuited for vibration patterns 434 that correspond to audio data in therange (0 hz to 20 hz). Similarly, haptic mechanism 446 and 448 may bebest suited for vibration patterns 436 and 438 corresponding to audiodata in the ranges (20 hz-40 hz) and (40 hz-200 hz), respectively. In amobile device with multiple haptic mechanisms, audio conversion tovibration pattern subsystem can create vibration patterns based on thespecific capabilities of the haptic mechanisms 444-448, thus allowingfor a richer variety of haptic feedback.

FIG. 5 is a block diagram illustrating playback of audio and hapticdata. Mobile device 500 may contain audio and haptic data files storedin combined audio and haptic data store 544. The audio and haptic datamay be part of separate files or combined into a single audio and hapticfile. Audio subsystem can receive data 520, which may contain both audioand haptic data, process, and communicate the audio data 520 tospeaker(s) 540 for audible playback to a user of mobile device 500.Haptic subsystem 514 can receive data 522, which may contain both audioand haptic data, process, and communicate the haptic data to hapticmechanism (s) 542 for actuating haptic feedback, e.g., a vibrationpattern.

FIG. 6 is a flowchart illustrating an exemplary method 600 forconverting audio data to haptic data. In step 610, a device may receiveaudio data corresponding to audible frequencies. For example, ambientaudio data received from a microphone in real-time, music audio datafrom data storage, or audio data input from another device. In step 620,a range of frequencies within the audio data may be converted intohaptic data. In step 630, a range of frequencies within the audio datamay be shifted into a different range of audible frequencies. In step640, the haptic and shifted audio data can be combined into a singledata file. In step 650, the combined haptic and audio data can beconveyed to a mobile device for audible and haptic feedback at themobile device.

FIG. 7 is a flowchart illustrating an exemplary method 700 forconverting audio data to haptic data. Method 700 can begin with audiodata 710. Audio data 710 may be filtered by audio data filter 712 inaccordance with hearing profile 718 to isolate a particular range ofaudible frequencies for frequency shifting. Frequency shifter 714 canshift the filtered audio data received from audio data filter 712 into adifferent frequency range as specified by hearing profile 718. Likewise,audio filter 720 can filter audio data 710 in accordance with hearingprofile 718 to isolate a particular range of audible frequencies forconversion into haptic data. Haptic converter 722 can convert thefiltered audio data received from audio data filter 720 into a hapticdata corresponding to a vibration pattern as specified by hearingprofile 718. Mixer 726 can receive shifted frequency data 716 and hapticdata 724 and combine the data into combined audio and haptic data 728.

Users of a mobile device may possess a wide variety of hearingimpairments. Therefore, it may be beneficial to provide a user of amobile device with the ability to create a hearing profile specificallytailored to the user. FIG. 8 is a flowchart illustrating an exemplarymethod 800 for creating a hearing profile. In step 810, a user selectsto begin creation of a hearing profile. In step 812, a test audio signalmay be communicated to the user. In step 814, the mobile device canreceive input from the user that specifies whether or not the user wasable to hear the audio signal. In step 816, the hearing profile isupdated to reflect whether or not the user was able to hear the audiblesignal in step 812. For example, the audible signal played in step 812may have been a frequency beyond the hearing capabilities of the user,in which case the hearing profile would be updated to reflect that theuser cannot hear that frequency. In step 818, either from user input orbased on a predetermined number of test audio signals, the hearingprofile creation process can play another test audio signal. Otherwise,the user may specify certain preferences to be included in the customhearing profile. For example, a user may prefer that sound effectscontained in audio data be converted to haptic data and sent to a hapticmechanism and that spoken tracks be frequency shifted and sent to anaudio output device. In step 820, the custom hearing profile creationprocess may ask a user more general questions, such as, “what type ofhearing impairment do you have,” “would you like to shift thesefrequencies down or up,” or “would you like to add haptic feedback toyour movie watching experience.” In step 822, the hearing profile may bestored in a suitable hearing profile data storage device.

In addition to or in place of creating a hearing profile, a user mayselect a hearing profile from a predetermined number of predefinedhearing profiles. For example, hearing profiles that have been createdbased on hearing impairment standards or conventions.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made. For example, whilethe examples described herein discuss audio and haptic feedback, in someexamples visual feedback may also be conveyed to a user of a mobiledevice. For example, visual feedback on a display of the mobile device,using lights on the mobile device, or any other suitable visual means.Audio data may be converted to visual data by utilizing similartechniques to those described in this disclosure for converting audiodata to haptic data.

In addition, other steps may be provided, or steps may be eliminated,from the described flows, and other components may be added to, orremoved from, the described systems. Accordingly, other implementationsare within the scope of the following claims.

1-42. (canceled)
 43. A method, comprising: at an electronic devicecomprising one or more audio output devices, one or more haptic outputdevices that are distinct from the one or more audio output devices, andone or more input devices: receiving, via the one or more input devices,a request to perform an operation associated with audio content; inresponse to receiving the request to perform the operation associatedwith audio content: retrieving audio feedback data associated with theaudio content and haptic feedback data associated with the audiocontent, wherein the haptic feedback data includes: a first hapticportion that is associated with a corresponding first audio portion ofthe audio feedback data, and a second haptic portion that is associatedwith a corresponding second audio portion of the second audio feedbackdata; outputting, via the one or more audio output devices, the firstaudio portion concurrently with outputting, via the one or more hapticoutput devices, the first haptic portion; and after outputting the firstaudio portion and the first haptic portion, outputting, via the one ormore audio output devices, the second audio portion concurrently withoutputting, via the one or more haptic output devices, the second hapticportion.
 44. The method of claim 43 wherein the audio feedback data andhaptic feedback data are stored as a single file.
 45. The method ofclaim 43 wherein the audio feedback data and the haptic feedback dataare stored as separate files that are associated with each other. 46.The method of claim 43 wherein the audio feedback data is mapped to arespective time sequence of a sequence of media content and the hapticfeedback data is mapped to the same time sequence.
 47. The method ofclaim 43 wherein the audio feedback and haptic feedback is coordinatedwith visual feedback displayed on a display of the device.
 48. Themethod of claim 43 wherein the haptic feedback comprises a vibrationbased on a stored vibration pattern.
 49. The method of claim 43 whereinthe haptic feedback comprises temperature variation.
 50. The method ofclaim 43 wherein the haptic feedback comprises electric stimulus to ahuman user.
 51. The method of claim 43 wherein at least one of the audiooutput devices comprises a speaker.
 52. The method of claim 43 whereinat least one of the haptic output devices comprises a spinning motor,servo motor, piezoelectric motor, or vibrator.
 53. An electronic devicecomprising: one or more audio output devices one or more haptic outputdevices that are distinct from the one or more audio output devices oneor more input devices; and a processor configured to executeinstructions, stored in a non-transitory machine-readable medium, tocause the electronic device to perform operations comprising: receiving,via the one or more input devices, a request to perform an operationassociated with audio content; in response to receiving the request toperform the operation associated with audio content: retrieving audiofeedback data associated with the audio content and haptic feedback dataassociated with the audio content, wherein the haptic feedback dataincludes: a first haptic portion that is associated with a correspondingfirst audio portion of the audio feedback data, and a second hapticportion that is associated with a corresponding second audio portion ofthe second audio feedback data; outputting, via the one or more audiooutput devices, the first audio portion concurrently with outputting,via the one or more haptic output devices, the first haptic portion; andafter outputting the first audio portion and the first haptic portion,outputting, via the one or more audio output devices, the second audioportion concurrently with outputting, via the one or more haptic outputdevices, the second haptic portion.
 54. The electronic device of claim53 wherein the audio feedback data and haptic feedback data are storedas a single file.
 55. The electronic device of claim 53 wherein theaudio feedback data and the haptic feedback data are stored as separatefiles that are associated with each other.
 56. The electronic device ofclaim 53 wherein the audio feedback data is mapped to a respective timesequence of a sequence of media content and the haptic feedback data ismapped to the same time sequence.
 57. The electronic device of claim 53wherein the audio feedback and haptic feedback is coordinated withvisual feedback displayed on a display of the device.
 58. The electronicdevice of claim 53 wherein the haptic feedback comprises a vibrationbased on a stored vibration pattern.
 59. The electronic device of claim53 wherein the haptic feedback comprises temperature variation.
 60. Theelectronic device of claim 53 wherein the haptic feedback compriseselectric stimulus to a human user.
 61. The electronic device of claim 53wherein at least one of the audio output devices comprises a speaker.62. The electronic device of claim 53 wherein at least one of the hapticoutput devices comprises a spinning motor, servo motor, piezoelectricmotor, or vibrator.
 63. A computer program product, encoded on anon-transitory computer-readable medium, operable to cause a dataprocessing apparatus to perform operations at an electronic devicecomprising one or more audio output devices, one or more haptic outputdevices that are distinct from the one or more audio output devices, andone or more input devices, the operations comprising: receiving, via theone or more input devices, a request to perform an operation associatedwith audio content; in response to receiving the request to perform theoperation associated with audio content: retrieving audio feedback dataassociated with the audio content and haptic feedback data associatedwith the audio content, wherein the haptic feedback data includes: afirst haptic portion that is associated with a corresponding first audioportion of the audio feedback data, and a second haptic portion that isassociated with a corresponding second audio portion of the second audiofeedback data; outputting, via the one or more audio output devices, thefirst audio portion concurrently with outputting, via the one or morehaptic output devices, the first haptic portion; and after outputtingthe first audio portion and the first haptic portion, outputting, viathe one or more audio output devices, the second audio portionconcurrently with outputting, via the one or more haptic output devices,the second haptic portion.
 64. The computer program product of claim 63wherein the audio feedback data and haptic feedback data are stored as asingle file.
 65. The computer program product of claim 63 wherein theaudio feedback data and the haptic feedback data are stored as separatefiles that are associated with each other.
 66. The computer programproduct of claim 63 wherein the audio feedback data is mapped to arespective time sequence of a sequence of media content and the hapticfeedback data is mapped to the same time sequence.
 67. The computerprogram product of claim 63 wherein the audio feedback and hapticfeedback is coordinated with visual feedback displayed on a display ofthe device.
 68. The computer program product of claim 63 wherein thehaptic feedback comprises a vibration based on a stored vibrationpattern.
 69. The computer program product of claim 63 wherein the hapticfeedback comprises temperature variation.
 70. The computer programproduct of claim 63 wherein the haptic feedback comprises electricstimulus to a human user.
 71. The computer program product of claim 63wherein at least one of the audio output devices comprises a speaker.72. The computer program product of claim 63 wherein at least one of thehaptic output devices comprises a spinning motor, servo motor,piezoelectric motor, or vibrator.